Using Vector Embeddings For Sentiment Analysis

Rod Acosta, Kevin Furbish, Ibrahim Khan, Anthony Washington

Introduction

  • What is NLP?
    • Natural Language Processing (NLP) is a field of artificial intelligence that focuses on the interaction between computers and human language. It involves teaching computers to understand, interpret, and respond to human language in a useful way.

Sentiment Analysis

  • Sentiment Analysis is a technique used in NLP to determine the emotional tone behind words. It helps to understand the sentiment expressed in text, whether it is positive, negative, or neutral.
  • Essential for understanding public opinion, aiding businesses in refining marketing strategies, improving products, and enhancing customer satisfaction.
  • The rise of social media and e-commerce platforms has increased the importance of sentiment analysis for real-time consumer behavior insights.
  • Hasan, Maliha, and Arifuzzaman (2019) used Twitter data to demonstrate the potential of sentiment analysis.
  • Their methodology combined Bag of Words (BoW) and Term Frequency-Inverse Document Frequency (TF-IDF) models with logistic regression, achieving an accuracy of 85.25% in classifying tweets as positive or negative (Hasan, Maliha, and Arifuzzaman 2019).

Sentiment Analysis (Continued)

  • Kathuria, Sethi, and Negi (2022) applied sentiment analysis to e-commerce reviews using various machine learning models (logistic regression, AdaBoost, SVM, Naive Bayes, and random forest).
  • Analyzed the Women’s E-commerce Clothing Reviews dataset to understand consumer behavior and the impact of electronic word-of-mouth (eWOM) on customer attitudes and product sales (Kathuria, Sethi, and Negi 2022).
  • Nasukawa and Yi (2003) focused on extracting sentiments linked to specific subjects rather than entire documents.
  • Their prototype system used a syntactic parser and sentiment lexicon to detect sentiments in web pages and news articles, offering detailed insights into specific opinions (Nasukawa and Yi 2003).
  • Conclusion:
    • These studies illustrate the evolution and application of sentiment analysis using NLP, highlighting its critical role in extracting insights from vast amounts of text data for strategic decision-making.

Vector Embeddings In Natural Language Processing

  • Vector Embeddings are a way to represent words or phrases in numerical form. Each word or phrase is converted into a vector (a list of numbers) which captures its meaning based on the context it is used in.

  • Vector embeddings transform text data into a numerical format that can be used by machine learning models. This numerical representation captures the semantic meaning of words and phrases.

  • Advantages of Embeddings:

    • Contextual Understanding: Captures the meaning of words based on their context.
    • Semantic Similarity: Words with similar meanings have similar vector representations.
    • Dimensionality Reduction: Reduces complex text data into simpler numerical form.
  • Advanced Embedding Models:
    • Sentence embeddings build on word embeddings to represent sentences, improving over Bag of Words models, which have high dimensionality and sparseness issues(Pilehvar and Camacho-Collados 2020).
    • Document embeddings further extend this concept to encode entire documents for NLP tasks where word ordering is important(Pilehvar and Camacho-Collados 2020).

Using Embeddings For Sentiment Analysis

  • Performance Improvement:
    • Word embedding and deep learning models have significantly enhanced NLP tasks, including sentiment analysis (Kasri et al. 2022).
    • Popular methods include Word2Vec and Global Vectors (GloVe), trained on different datasets (Google News and Wikipedia 2014, respectively) (N et al. 2024).
  • Challenges with Pre-trained Embeddings:
    • Pre-trained word embeddings often fail to capture contextual sentiment information, leading to inaccuracies (e.g., mapping “good” and “bad” to neighboring vectors) [Liang et al. (2023)](Tang et al. 2016).
    • The paper aims to explore these challenges and investigate alternative embedding methods for more accurate sentiment analysis.

Methods

What is a neural Network?

  • A neural network is a type of algorithm that mimics the structure and function of the human brain. Their goal is to create an artificial system that can process and analyze data in a similar way.
  • There are different types of neural networks but there are some common elements between most of them. Those elements are:
    • Artificial Neurons
    • Layers

Neural Network Layers

  • Neural networks usually have three types of layers:
    • Input Layer
    • Hidden layers
    • Output layer

What are embeddings?

  • Embeddings are a technique that allow us to map words or phrases into a corresponding vector of real numbers, where the position and direction of the vector capture the word’s semantic meaning in relation to other words.
  • They make high-dimensional data like words readable to our algorithm/model and allows our model to recognize and learn meaningful relationships and similarities between words

Dense Layer & Cosine Similarity

  • Cosine Similarity
    • Measures the cosine of the angle between two non-zero vectors, providing a measure of similarity.
    • The smaller the angle the higher the similarity between the two vectors.
    • \(cosine\_similarity(u,v) = \frac{u.v}{||u|| ||v||}\)
  • Dense Layer
    • A logistic regression model with a sigmoid activation function used for binary classification.
    • It outputs the probability that the input belongs to a positive class.
    • \(y=\sigma(W⋅z+b)\)
    • Where:
      • z is the flattened input vector.
      • W is the weight vector.
      • b is the bias term.
      • \(\sigma(x) = \frac{1}{1+e^{-x}}\) is the sigmoid function.

Sentiment Analysis

  • Through the use of a neural network and it’s hidden layers (embedding & dense), and the cosine similarity we are able to take inputs and classify them as being part of a positive or negative class based on what our model has learned from our training dataset.

Analysis and Results

Dataset Description

  • Data set of 25,000 movie reviews from IMDB
  • Max 30 reviews for each movie since popular movies are rated more often than unpopular movies.
  • Includes only the top 5,000 most frequent words, minus the top 50 most frequent words
  • IMDB reviews 1-10 star rating converted to a 0-1 scale

Vectorization

  • Neural Networks require numeric inputs, not the natural language of reviews
  • Vectorization represents each word with a unique numeric substitution. For example the following texts:
    • “this is fun”, “fun times ahead”, “fun is ahead of times”
  • Results in a vocabulary of [this, is, fun, times, ahead, of]
  • Vectorized observations: [1, 2, 3], [3, 4, 5] and [3, 2, 5, 6, 4]
  • TensorFlow (TF) includes the IMDB dataset in a vectorized format

Overview of Embeddings Neural Network

  • Neural network model implemented and trained using TF’s Keras package
  • TF and Keras Python implementations accessed from R via Reticulate package
  • Model learns embeddings for each word in the vocabulary
  • Multi-dimensional embedding vectors placed close in the learned vector-space to vectors of similar words
  • Similar means having a similar contextual meaning in the training dataset and its sentiment classification
    • “gem” and “favorite” would be similar in context of a movie review, but not in a general context

Neural Network Implementation In Keras

  • Keras Sequential model (model is built layer-by-layer)
  • The neural networks leverages the Keras embedding layer
    • Converts input vocabulary index into a vector of a chosen dimension
    • Dimensionality important hyperparameter that controls compression vs overfitting
  • Output of embedding layer is the same as the dimensionality hyperparameter
  • Embedding layer connects to a dense layer
  • Dense layers require a 1D input vector so embedding layer output is flattened to 1D
  • Dense layer has a 1 output unit with sigmoid activation function that receives the flattened embedding layer output
  • Output unit is the sentiment score
  • Back-propagation trains embedding layer weights to be similar based on sentiment

Determining Vector Dimensionality

  • Number of dimensions for the embeddings must be selected
  • Dimensionality selection often done ad hoc with a functionality test
  • We tested 2 through 7 dimensions
  • Models compared by testing accuracy and qualitatievly looking at closest words

Model Performance: 2D

  • 2 dimensions had best accuracy, but failed to capture meaning
  • Closest embeddings to the embedding for “awful”:
word cosine similarity
awful 1.0
lame 1.0
alcoholic 1.0
sadly 0.99
relevant 0.99
are 0.99

Model Performance: 2D vs 7D

  • 7 Dimensions had similar accuracy to 2, but better embedding performance
Table 1: Closest Words By Dimensionality
(a) 2 Dimensions
reference word closest words
awful lame, alcoholic, sadly, relevant, are
mediocre effort, turkey, terrible, stereotype, repeat
perfect lovers, sing, manager, bath, donald
favorite deeply, roud, marie, polanski, poetry
(b) 7 Dimensions
reference word closest words
awful ultimately, painful, sorry, fake, nowhere
mediocre teeth, incompetent, main, disappointing, generous
perfect great, seeking, freedom, tremendous, excellent
favorite paulie, excellent, necessary, great, seeking

Data Visualization

  • The Umap below shows the location of where all the words are embedded on a 2D plane. The words of the IMDB dataset are clustered densely around the origin of the graph.

Data Visualization

  • The partial Umap below zooms in to the center of the Umap where it is most dense and labels the words to display all the words that are located near each other based on the embedding.

Data Visualization

  • The Umap below shows the 25 most similar words to the word “angry” according to the embedding.

Conclusion

  • The development and analysis of the word embedding model for classifying IMDB movie reviews demonstrated promising results.

  • The optimal number of embedding dimensions was identified as 7, achieving an accuracy of 87.34% on the test data set.

  • This was determined through extensive experimentation, revealing that higher dimensions, such as 7, provided competitive and consistent accuracy.

Conclusion

  • The model’s performance is noteworthy, given the constraints of training on only the top 5000 most common words, minimal data preprocessing, and limiting input sequences to the first 500 words of each review.

  • These factors illustrate the model’s robustness and effectiveness in capturing the semantic relationships within the data.

Conclusion

  • The embedding similarity results showed that the model could meaningfully capture semantic relationships, as evidenced by the coherent and relevant similar words found for terms like “awful,” “mediocre,” “perfect,” and “favorite.” The final training session, capped at 6 epochs, ensured the model did not overfit, maintaining its accuracy and reliability.

  • Overall, the model’s strong performance under constrained conditions highlights its potential for practical applications in sentiment analysis, offering an efficient and effective solution for understanding and categorizing movie reviews.

References

Hasan, Md. Rakibul, Maisha Maliha, and M. Arifuzzaman. 2019. “Sentiment Analysis with NLP on Twitter Data.” In 2019 International Conference on Computer, Communication, Chemical, Materials and Electronic Engineering (IC4ME2), 1–4. https://doi.org/10.1109/IC4ME247184.2019.9036670.
Kasri, Mohammed, Marouane Birjali, Mohamed Nabil, Abderrahim Beni-Hssane, Anas El-Ansari, and Mohamed El Fissaoui. 2022. “Refining Word Embeddings with Sentiment Information for Sentiment Analysis.” Journal of ICT Standardization 10 (3): 353–82. https://doi.org/10.13052/jicts2245-800X.1031.
Kathuria, Priyanshi, Parth Sethi, and Rithwick Negi. 2022. “Sentiment Analysis on e-Commerce Reviews and Ratings Using ML & NLP Models to Understand Consumer Behavior.” In 2022 International Conference on Recent Trends in Microelectronics, Automation, Computing and Communications Systems (ICMACC), 1–5. https://doi.org/10.1109/ICMACC54824.2022.10093674.
Liang, Bin, Rongdi Yin, Jiachen Du, Lin Gui, Yulan He, Min Yang, and Ruifeng Xu. 2023. “Embedding Refinement Framework for Targeted Aspect-Based Sentiment Analysis.” IEEE Transactions on Affective Computing 14 (1): 279–93. https://doi.org/10.1109/TAFFC.2021.3071388.
N, Lavanya B., Anitha Rathnam K. V, Kiran K, P. Deepa Shenoy, and Venugopal K. R. 2024. “Fusion of Deep Learning with Advanced and Traditional Embeddings in Sentiment Analysis.” In 2024 IEEE 9th International Conference for Convergence in Technology (I2CT), 1–6. https://doi.org/10.1109/I2CT61223.2024.10543279.
Nasukawa, Tetsuya, and Jeonghee Yi. 2003. “Sentiment Analysis: Capturing Favorability Using Natural Language Processing.” In Proceedings of the 2nd International Conference on Knowledge Capture, 70–77.
Pilehvar, Mohammad Taher, and Jose Camacho-Collados. 2020. Embeddings in Natural Language Processing: Theory and Advances in Vector Representations of Meaning. Morgan & Claypool Publishers.
Tang, Duyu, Furu Wei, Bing Qin, Nan Yang, Ting Liu, and Ming Zhou. 2016. “Sentiment Embeddings with Applications to Sentiment Analysis.” IEEE Transactions on Knowledge and Data Engineering 28 (2): 496–509. https://doi.org/10.1109/TKDE.2015.2489653.